Methods and systems that use feedback to distribute and manage alerts

ABSTRACT

The current document is directed to methods and systems that employ user feedback and user-initiated alert-generation-and-distribution-system modifications to provide for flexibility and responsiveness. In a described implementation, participants in a social environment provided by a collaborative alert-generation-and-distribution-system suggest modifications which are automatically or semi-automatically incorporated in an alert-generation-and-distribution-system when the participants agree to the suggested modifications.

TECHNICAL FIELD

The current document is directed to the generation and distribution ofalerts and, in particular, to methods and systems that employ userfeedback and user-initiated proposals from alert recipients to add,remove, and update information employed by an alert generation anddistribution system for alert generation and distribution.

BACKGROUND

During the past seven decades, electronic computing has evolved fromprimitive, vacuum-tube-based computer systems, initially developedduring the 1940s, to modern electronic computing systems in which largenumbers of multi-processor servers, work stations, and other individualcomputing systems are networked together with large-capacitydata-storage devices and other electronic devices to producegeographically distributed computing systems with hundreds of thousands,millions, or more components that provide enormous computationalbandwidths and data-storage capacities. These large, distributedcomputing systems are made possible by advances in computer networking,distributed operating systems and applications, data-storage appliances,computer hardware, and software technologies. Despite all of theseadvances, however, the rapid increase in the size and complexity ofcomputing systems has been accompanied by numerous scaling issues andtechnical challenges, including technical challenges associated withcommunications overheads encountered in parallelizing computationaltasks among multiple processors, component failures, anddistributed-system management. As new distributed-computing technologiesare developed and as general hardware and software technologies continueto advance, the current trend towards ever-larger and more complexdistributed computing systems appears likely to continue well into thefuture.

In modern computing systems, individual computers, subsystems, andcomponents generally output large volumes of status, informational, anderror messages that are collectively referred to, in the currentdocument, as “event messages.” In large, distributed computing systems,terabytes of event messages may be generated each day. The eventmessages are often collected into event logs stored as files indata-storage appliances and are often analyzed both in real time, asthey are generated and received, as well as retrospectively, after theevent messages have been initially processed and stored in event logs.Event messages may contain information that can be used to detectserious failures and operational deficiencies prior to the accumulationof a sufficient number of failures and system-degrading events that leadto data loss and significant down time. The information contained inevent messages may also be used to detect and ameliorate various typesof security breaches and issues, to intelligently manage and maintaindistributed computing systems, and to diagnose many different classes ofoperational problems, hardware-design deficiencies, and software-designdeficiencies.

In many systems, alerts are generated when certain types of eventmessages are received by monitoring-and-management systems. The alertsare distributed to personnel responsible for monitoring, managing, andadministering the systems, so that failures and system-degrading eventsare quickly evaluated and addressed. The alerts may be received anddisplayed on personal computers, laptops, and works stations, but mayalso to received and displayed on smart phone, tablets, and other typesof devices. Alerts may also be distributed to pagers, telephones, andother devices that receive alerts and notify personnel who own and/oruse the devices. Although alert distribution is an effective method forquickly notifying personnel and marshalling needed personnel to addresssystem failures and system-degrading events, currently available alertgeneration and distribution systems lack flexibility and responsivenessto user feedback. For example, it may turn out that a particularhigh-priority alert is often spuriously generated. In such cases. itwould be beneficial for the high-priority alert to be downgraded inpriority or disabled altogether, to avoid unnecessary diversion ofpersonnel to respond to spurious alerts. However, in currently availablesystems, such changes often require high-latency report submission,authorization, and reprogramming or reconfiguration, as a result ofwhich the spurious alert may continue to be generated for days or weeksbefore the alert can be disabled or reprogrammed. Users ofalert-generating systems and subsystems continue to seek more flexiblyand easily modified and managed alert systems.

SUMMARY

The current document is directed to methods and systems that employ userfeedback and user-initiated alert-generation-and-distribution-systemmodifications to provide for flexibility and responsiveness. In adescribed implementation, participants in a social environment providedby a collaborative alert-generation-and-distribution-system suggestmodifications which are automatically or semi-automatically incorporatedin an alert-generation-and-distribution-system when the participantsagree to the suggested modifications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a general architectural diagram for various types ofcomputers.

FIG. 2 illustrates an Internet-connected distributed computer system.

FIG. 3 illustrates cloud computing. In the recently developedcloud-computing paradigm, computing cycles and data-storage facilitiesare provided to organizations and individuals by cloud-computingproviders.

FIG. 4 illustrates generalized hardware and software components of ageneral-purpose computer system, such as a general-purpose computersystem having an architecture similar to that shown in FIG. 1.

FIGS. 5A-B illustrate two types of virtual machine and virtual-machineexecution environments.

FIG. 6 illustrates an OVF package.

FIG. 7 illustrates virtual data centers provided as an abstraction ofunderlying physical-data-center hardware components.

FIG. 8 illustrates virtual-machine components of a virtual-data-centermanagement server and physical servers of a physical data center abovewhich a virtual-data-center interface is provided by thevirtual-data-center management server.

FIG. 9 illustrates a cloud-director level of abstraction. In FIG. 9,three different physical data centers 902-904 are shown below planesrepresenting the cloud-director layer of abstraction 906-908.

FIG. 10 illustrates virtual-cloud-connector nodes (“VCC nodes”) and aVCC server, components of a distributed system that provides multi-cloudaggregation and that includes a cloud-connector server andcloud-connector nodes that cooperate to provide services that aredistributed across multiple clouds.

FIG. 11 illustrates a simple example of event-message logging andanalysis.

FIG. 12 shows a small, 11-entry portion of a log file from a distributedcomputer system.

FIG. 13 illustrates the distribution of alerts from a distributedcomputer system to system-administration-personnel devices.

FIG. 14 illustrates the display of an alert message on asystem-administration-personnel device.

FIGS. 15A-B provide control-flow diagrams that illustrate operation ofthe monitoring-and-management system shown in FIGS. 11 and 13.

FIGS. 16A-B illustrate, from a user-interface perspective, certainfeatures of the currently disclosed, collaborativealert-generation-and-distribution system or subsystem.

FIGS. 17A-B show a number of relational-database tables that representthe stored data that is maintained by the collaborativealert-generation-and-distribution system and that is used for alertdistribution, proposed-modification submission, voting for or againstproposed modifications, and other functionalities and facilitiesprovided by the collaborative alert-generation-and-distribution system.

FIG. 18 provides a number of examples of SQL queries in the context ofthe relational database tables shown in FIGS. 17A-B.

FIGS. 19A-F provide control-flow diagrams that illustrate portions of animplementation of the currently disclosed collaborativealert-generation-and-distribution system.

DETAILED DESCRIPTION

The current document is directed to methods and systems that employuser-initiated proposals and automated consensus determination todynamically modify the alert-generation-and-distribution systems. In afirst subsection, below, a detailed description of computer hardware,complex computational systems, and virtualization is provided withreference to FIGS. 1-10. In a second subsection, event-messageprocessing and alert generation and distribution are discussed withreference to FIGS. 11-15B. A final subsection discusses methods andalert-generation-and-distribution systems

Computer Hardware, Complex Computational Systems, and Virtualization

The term “abstraction” is not, in any way, intended to mean or suggestan abstract idea or concept. Computational abstractions are tangible,physical interfaces that are implemented, ultimately, using physicalcomputer hardware, data-storage devices, and communications systems.Instead, the term “abstraction” refers, in the current discussion, to alogical level of functionality encapsulated within one or more concrete,tangible, physically-implemented computer systems with definedinterfaces through which electronically-encoded data is exchanged,process execution launched, and electronic services are provided.Interfaces may include graphical and textual data displayed on physicaldisplay devices as well as computer programs and routines that controlphysical computer processors to carry out various tasks and operationsand that are invoked through electronically implemented applicationprogramming interfaces (“APIs”) and other electronically implementedinterfaces. There is a tendency among those unfamiliar with moderntechnology and science to misinterpret the terms “abstract” and“abstraction,” when used to describe certain aspects of moderncomputing. For example, one frequently encounters assertions that,because a computational system is described in terms of abstractions,functional layers, and interfaces, the computational system is somehowdifferent from a physical machine or device. Such allegations areunfounded. One only needs to disconnect a computer system or group ofcomputer systems from their respective power supplies to appreciate thephysical, machine nature of complex computer technologies. One alsofrequently encounters statements that characterize a computationaltechnology as being “only software,” and thus not a machine or device.Software is essentially a sequence of encoded symbols, such as aprintout of a computer program or digitally encoded computerinstructions sequentially stored in a file on an optical disk or withinan electromechanical mass-storage device. Software alone can do nothing.It is only when encoded computer instructions are loaded into anelectronic memory within a computer system and executed on a physicalprocessor that so-called “software implemented” functionality isprovided. The digitally encoded computer instructions are an essentialand physical control component of processor-controlled machines anddevices, no less essential and physical than a cam-shaft control systemin an internal-combustion engine. Multi-cloud aggregations,cloud-computing services, virtual-machine containers and virtualmachines, communications interfaces, and many of the other topicsdiscussed below are tangible, physical components of physical,electro-optical-mechanical computer systems.

FIG. 1 provides a general architectural diagram for various types ofcomputers. Computers that receive, process, and store event messages maybe described by the general architectural diagram shown in FIG. 1, forexample. The computer system contains one or multiple central processingunits (“CPUs”) 102-105, one or more electronic memories 108interconnected with the CPUs by a CPU/memory-subsystem bus 110 ormultiple busses, a first bridge 112 that interconnects theCPU/memory-subsystem bus 110 with additional busses 114 and 116, orother types of high-speed interconnection media, including multiple,high-speed serial interconnects. These busses or serialinterconnections, in turn, connect the CPUs and memory with specializedprocessors, such as a graphics processor 118, and with one or moreadditional bridges 120, which are interconnected with high-speed seriallinks or with multiple controllers 122-127, such as controller 127, thatprovide access to various different types of mass-storage devices 128,electronic displays, input devices, and other such components,subcomponents, and computational resources. It should be noted thatcomputer-readable data-storage devices include optical andelectromagnetic disks, electronic memories, and other physicaldata-storage devices. Those familiar with modern science and technologyappreciate that electromagnetic radiation and propagating signals do notstore data for subsequent retrieval, and can transiently “store” only abyte or less of information per mile, far less information than neededto encode even the simplest of routines.

Of course, there are many different types of computer-systemarchitectures that differ from one another in the number of differentmemories, including different types of hierarchical cache memories, thenumber of processors and the connectivity of the processors with othersystem components, the number of internal communications busses andserial links, and in many other ways. However, computer systemsgenerally execute stored programs by fetching instructions from memoryand executing the instructions in one or more processors. Computersystems include general-purpose computer systems, such as personalcomputers (“PCs”), various types of servers and workstations, andhigher-end mainframe computers, but may also include a plethora ofvarious types of special-purpose computing devices, includingdata-storage systems, communications routers, network nodes, tabletcomputers, and mobile telephones.

FIG. 2 illustrates an Internet-connected distributed computer system. Ascommunications and networking technologies have evolved in capabilityand accessibility, and as the computational bandwidths, data-storagecapacities, and other capabilities and capacities of various types ofcomputer systems have steadily and rapidly increased, much of moderncomputing now generally involves large distributed systems and computersinterconnected by local networks, wide-area networks, wirelesscommunications, and the Internet. FIG. 2 shows a typical distributedsystem in which a large number of PCs 202-205, a high-end distributedmainframe system 210 with a large data-storage system 212, and a largecomputer center 214 with large numbers of rack-mounted servers or bladeservers all interconnected through various communications and networkingsystems that together comprise the Internet 216. Such distributedcomputing systems provide diverse arrays of functionalities. Forexample, a PC user sitting in a home office may access hundreds ofmillions of different web sites provided by hundreds of thousands ofdifferent web servers throughout the world and may accesshigh-computational-bandwidth computing services from remote computerfacilities for running complex computational tasks.

Until recently, computational services were generally provided bycomputer systems and data centers purchased, configured, managed, andmaintained by service-provider organizations. For example, an e-commerceretailer generally purchased, configured, managed, and maintained a datacenter including numerous web servers, back-end computer systems, anddata-storage systems for serving web pages to remote customers,receiving orders through the web-page interface, processing the orders,tracking completed orders, and other myriad different tasks associatedwith an e-commerce enterprise.

FIG. 3 illustrates cloud computing. In the recently developedcloud-computing paradigm, computing cycles and data-storage facilitiesare provided to organizations and individuals by cloud-computingproviders. In addition, larger organizations may elect to establishprivate cloud-computing facilities in addition to, or instead of,subscribing to computing services provided by public cloud-computingservice providers. In FIG. 3, a system administrator for anorganization, using a PC 302, accesses the organization's private cloud304 through a local network 306 and private-cloud interface 308 and alsoaccesses, through the Internet 310, a public cloud 312 through apublic-cloud services interface 314. The administrator can, in eitherthe case of the private cloud 304 or public cloud 312, configure virtualcomputer systems and even entire virtual data centers and launchexecution of application programs on the virtual computer systems andvirtual data centers in order to carry out any of many different typesof computational tasks. As one example, a small organization mayconfigure and run a virtual data center within a public cloud thatexecutes web servers to provide an e-commerce interface through thepublic cloud to remote customers of the organization, such as a userviewing the organization's e-commerce web pages on a remote user system316.

Cloud-computing facilities are intended to provide computationalbandwidth and data-storage services much as utility companies provideelectrical power and water to consumers. Cloud computing providesenormous advantages to small organizations without the resources topurchase, manage, and maintain in-house data centers. Such organizationscan dynamically add and delete virtual computer systems from theirvirtual data centers within public clouds in order to trackcomputational-bandwidth and data-storage needs, rather than purchasingsufficient computer systems within a physical data center to handle peakcomputational-bandwidth and data-storage demands. Moreover, smallorganizations can completely avoid the overhead of maintaining andmanaging physical computer systems, including hiring and periodicallyretraining information-technology specialists and continuously payingfor operating-system and database-management-system upgrades.Furthermore, cloud-computing interfaces allow for easy andstraightforward configuration of virtual computing facilities,flexibility in the types of applications and operating systems that canbe configured, and other functionalities that are useful even for ownersand administrators of private cloud-computing facilities used by asingle organization.

FIG. 4 illustrates generalized hardware and software components of ageneral-purpose computer system, such as a general-purpose computersystem having an architecture similar to that shown in FIG. 1. Thecomputer system 400 is often considered to include three fundamentallayers: (1) a hardware layer or level 402; (2) an operating-system layeror level 404; and (3) an application-program layer or level 406. Thehardware layer 402 includes one or more processors 408, system memory410, various different types of input-output (“I/O”) devices 410 and412, and mass-storage devices 414. Of course, the hardware level alsoincludes many other components, including power supplies, internalcommunications links and busses, specialized integrated circuits, manydifferent types of processor-controlled or microprocessor-controlledperipheral devices and controllers, and many other components. Theoperating system 404 interfaces to the hardware level 402 through alow-level operating system and hardware interface 416 generallycomprising a set of non-privileged computer instructions 418, a set ofprivileged computer instructions 420, a set of non-privileged registersand memory addresses 422, and a set of privileged registers and memoryaddresses 424. In general, the operating system exposes non-privilegedinstructions, non-privileged registers, and non-privileged memoryaddresses 426 and a system-call interface 428 as an operating-systeminterface 430 to application programs 432-436 that execute within anexecution environment provided to the application programs by theoperating system. The operating system, alone, accesses the privilegedinstructions, privileged registers, and privileged memory addresses. Byreserving access to privileged instructions, privileged registers, andprivileged memory addresses, the operating system can ensure thatapplication programs and other higher-level computational entitiescannot interfere with one another's execution and cannot change theoverall state of the computer system in ways that could deleteriouslyimpact system operation. The operating system includes many internalcomponents and modules, including a scheduler 442, memory management444, a file system 446, device drivers 448, and many other componentsand modules. To a certain degree, modern operating systems providenumerous levels of abstraction above the hardware level, includingvirtual memory, which provides to each application program and othercomputational entities a separate, large, linear memory-address spacethat is mapped by the operating system to various electronic memoriesand mass-storage devices. The scheduler orchestrates interleavedexecution of various different application programs and higher-levelcomputational entities, providing to each application program a virtual,stand-alone system devoted entirely to the application program. From theapplication program's standpoint, the application program executescontinuously without concern for the need to share processor resourcesand other system resources with other application programs andhigher-level computational entities. The device drivers abstract detailsof hardware-component operation, allowing application programs to employthe system-call interface for transmitting and receiving data to andfrom communications networks, mass-storage devices, and other I/Odevices and subsystems. The file system 436 facilitates abstraction ofmass-storage-device and memory resources as a high-level,easy-to-access, file-system interface. Thus, the development andevolution of the operating system has resulted in the generation of atype of multi-faceted virtual execution environment for applicationprograms and other higher-level computational entities.

While the execution environments provided by operating systems haveproved to be an enormously successful level of abstraction withincomputer systems, the operating-system-provided level of abstraction isnonetheless associated with difficulties and challenges for developersand users of application programs and other higher-level computationalentities. One difficulty arises from the fact that there are manydifferent operating systems that run within various different types ofcomputer hardware. In many cases, popular application programs andcomputational systems are developed to run on only a subset of theavailable operating systems, and can therefore be executed within only asubset of the various different types of computer systems on which theoperating systems are designed to run. Often, even when an applicationprogram or other computational system is ported to additional operatingsystems, the application program or other computational system cannonetheless run more efficiently on the operating systems for which theapplication program or other computational system was originallytargeted. Another difficulty arises from the increasingly distributednature of computer systems. Although distributed operating systems arethe subject of considerable research and development efforts, many ofthe popular operating systems are designed primarily for execution on asingle computer system. In many cases, it is difficult to moveapplication programs, in real time, between the different computersystems of a distributed computer system for high-availability,fault-tolerance, and load-balancing purposes. The problems are evengreater in heterogeneous distributed computer systems which includedifferent types of hardware and devices running different types ofoperating systems. Operating systems continue to evolve, as a result ofwhich certain older application programs and other computationalentities may be incompatible with more recent versions of operatingsystems for which they are targeted, creating compatibility issues thatare particularly difficult to manage in large distributed systems.

For all of these reasons, a higher level of abstraction, referred to asthe “virtual machine,” has been developed and evolved to furtherabstract computer hardware in order to address many difficulties andchallenges associated with traditional computing systems, including thecompatibility issues discussed above. FIGS. 5A-B illustrate two types ofvirtual machine and virtual-machine execution environments. FIGS. 5A-Buse the same illustration conventions as used in FIG. 4. FIG. 5A shows afirst type of virtualization. The computer system 500 in FIG. 5Aincludes the same hardware layer 502 as the hardware layer 402 shown inFIG. 4. However, rather than providing an operating system layerdirectly above the hardware layer, as in FIG. 4, the virtualizedcomputing environment illustrated in FIG. 5A features a virtualizationlayer 504 that interfaces through a virtualization-layer/hardware-layerinterface 506, equivalent to interface 416 in FIG. 4, to the hardware.The virtualization layer provides a hardware-like interface 508 to anumber of virtual machines, such as virtual machine 510, executing abovethe virtualization layer in a virtual-machine layer 512. Each virtualmachine includes one or more application programs or other higher-levelcomputational entities packaged together with an operating system,referred to as a “guest operating system,” such as application 514 andguest operating system 516 packaged together within virtual machine 510.Each virtual machine is thus equivalent to the operating-system layer404 and application-program layer 406 in the general-purpose computersystem shown in FIG. 4. Each guest operating system within a virtualmachine interfaces to the virtualization-layer interface 508 rather thanto the actual hardware interface 506. The virtualization layerpartitions hardware resources into abstract virtual-hardware layers towhich each guest operating system within a virtual machine interfaces.The guest operating systems within the virtual machines, in general, areunaware of the virtualization layer and operate as if they were directlyaccessing a true hardware interface. The virtualization layer ensuresthat each of the virtual machines currently executing within the virtualenvironment receive a fair allocation of underlying hardware resourcesand that all virtual machines receive sufficient resources to progressin execution. The virtualization-layer interface 508 may differ fordifferent guest operating systems. For example, the virtualization layeris generally able to provide virtual hardware interfaces for a varietyof different types of computer hardware. This allows, as one example, avirtual machine that includes a guest operating system designed for aparticular computer architecture to run on hardware of a differentarchitecture. The number of virtual machines need not be equal to thenumber of physical processors or even a multiple of the number ofprocessors.

The virtualization layer includes a virtual-machine-monitor module 518(“VMM”) that virtualizes physical processors in the hardware layer tocreate virtual processors on which each of the virtual machinesexecutes. For execution efficiency, the virtualization layer attempts toallow virtual machines to directly execute non-privileged instructionsand to directly access non-privileged registers and memory. However,when the guest operating system within a virtual machine accessesvirtual privileged instructions, virtual privileged registers, andvirtual privileged memory through the virtualization-layer interface508, the accesses result in execution of virtualization-layer code tosimulate or emulate the privileged resources. The virtualization layeradditionally includes a kernel module 520 that manages memory,communications, and data-storage machine resources on behalf ofexecuting virtual machines (“VM kernel”). The VM kernel, for example,maintains shadow page tables on each virtual machine so thathardware-level virtual-memory facilities can be used to process memoryaccesses. The VM kernel additionally includes routines that implementvirtual communications and data-storage devices as well as devicedrivers that directly control the operation of underlying hardwarecommunications and data-storage devices. Similarly, the VM kernelvirtualizes various other types of I/O devices, including keyboards,optical-disk drives, and other such devices. The virtualization layeressentially schedules execution of virtual machines much like anoperating system schedules execution of application programs, so thatthe virtual machines each execute within a complete and fully functionalvirtual hardware layer.

FIG. 5B illustrates a second type of virtualization. In FIG. 5B, thecomputer system 540 includes the same hardware layer 542 and softwarelayer 544 as the hardware layer 402 shown in FIG. 4. Several applicationprograms 546 and 548 are shown running in the execution environmentprovided by the operating system. In addition, a virtualization layer550 is also provided, in computer 540, but, unlike the virtualizationlayer 504 discussed with reference to FIG. 5A, virtualization layer 550is layered above the operating system 544, referred to as the “host OS,”and uses the operating system interface to accessoperating-system-provided functionality as well as the hardware. Thevirtualization layer 550 comprises primarily a VMM and a hardware-likeinterface 552, similar to hardware-like interface 508 in FIG. 5A. Thevirtualization-layer/hardware-layer interface 552, equivalent tointerface 416 in FIG. 4, provides an execution environment for a numberof virtual machines 556-558, each including one or more applicationprograms or other higher-level computational entities packaged togetherwith a guest operating system.

In FIGS. 5A-B, the layers are somewhat simplified for clarity ofillustration. For example, portions of the virtualization layer 550 mayreside within the host-operating-system kernel, such as a specializeddriver incorporated into the host operating system to facilitatehardware access by the virtualization layer.

It should be noted that virtual hardware layers, virtualization layers,and guest operating systems are all physical entities that areimplemented by computer instructions stored in physical data-storagedevices, including electronic memories, mass-storage devices, opticaldisks, magnetic disks, and other such devices. The term “virtual” doesnot, in any way, imply that virtual hardware layers, virtualizationlayers, and guest operating systems are abstract or intangible. Virtualhardware layers, virtualization layers, and guest operating systemsexecute on physical processors of physical computer systems and controloperation of the physical computer systems, including operations thatalter the physical states of physical devices, including electronicmemories and mass-storage devices. They are as physical and tangible asany other component of a computer since, such as power supplies,controllers, processors, busses, and data-storage devices.

A virtual machine or virtual application, described below, isencapsulated within a data package for transmission, distribution, andloading into a virtual-execution environment. One public standard forvirtual-machine encapsulation is referred to as the “open virtualizationformat” (“OVF”). The OVF standard specifies a format for digitallyencoding a virtual machine within one or more data files. FIG. 6illustrates an OVF package. An OVF package 602 includes an OVFdescriptor 604, an OVF manifest 606, an OVF certificate 608, one or moredisk-image files 610-611, and one or more resource files 612-614. TheOVF package can be encoded and stored as a single file or as a set offiles. The OVF descriptor 604 is an XML document 620 that includes ahierarchical set of elements, each demarcated by a beginning tag and anending tag. The outermost, or highest-level, element is the envelopeelement, demarcated by tags 622 and 623. The next-level element includesa reference element 626 that includes references to all files that arepart of the OVF package, a disk section 628 that contains metainformation about all of the virtual disks included in the OVF package,a networks section 630 that includes meta information about all of thelogical networks included in the OVF package, and a collection ofvirtual-machine configurations 632 which further includes hardwaredescriptions of each virtual machine 634. There are many additionalhierarchical levels and elements within a typical OVF descriptor. TheOVF descriptor is thus a self-describing, XML file that describes thecontents of an OVF package. The OVF manifest 606 is a list ofcryptographic-hash-function-generated digests 636 of the entire OVFpackage and of the various components of the OVF package. The OVFcertificate 608 is an authentication certificate 640 that includes adigest of the manifest and that is cryptographically signed. Disk imagefiles, such as disk image file 610, are digital encodings of thecontents of virtual disks and resource files 612 are digitally encodedcontent, such as operating-system images. A virtual machine or acollection of virtual machines encapsulated together within a virtualapplication can thus be digitally encoded as one or more files within anOVF package that can be transmitted, distributed, and loaded usingwell-known tools for transmitting, distributing, and loading files. Avirtual appliance is a software service that is delivered as a completesoftware stack installed within one or more virtual machines that isencoded within an OVF package.

The advent of virtual machines and virtual environments has alleviatedmany of the difficulties and challenges associated with traditionalgeneral-purpose computing. Machine and operating-system dependencies canbe significantly reduced or entirely eliminated by packagingapplications and operating systems together as virtual machines andvirtual appliances that execute within virtual environments provided byvirtualization layers running on many different types of computerhardware. A next level of abstraction, referred to as virtual datacenters or virtual infrastructure, provide a data-center interface tovirtual data centers computationally constructed within physical datacenters. FIG. 7 illustrates virtual data centers provided as anabstraction of underlying physical-data-center hardware components. InFIG. 7, a physical data center 702 is shown below a virtual-interfaceplane 704. The physical data center consists of a virtual-data-centermanagement server 706 and any of various different computers, such asPCs 708, on which a virtual-data-center management interface may bedisplayed to system administrators and other users. The physical datacenter additionally includes generally large numbers of servercomputers, such as server computer 710, that are coupled together bylocal area networks, such as local area network 712 that directlyinterconnects server computer 710 and 714-720 and a mass-storage array722. The physical data center shown in FIG. 7 includes three local areanetworks 712, 724, and 726 that each directly interconnects a bank ofeight servers and a mass-storage array. The individual server computers,such as server computer 710, each includes a virtualization layer andruns multiple virtual machines. Different physical data centers mayinclude many different types of computers, networks, data-storagesystems and devices connected according to many different types ofconnection topologies. The virtual-data-center abstraction layer 704, alogical abstraction layer shown by a plane in FIG. 7, abstracts thephysical data center to a virtual data center comprising one or moreresource pools, such as resource pools 730-732, one or more virtual datastores, such as virtual data stores 734-736, and one or more virtualnetworks. In certain implementations, the resource pools abstract banksof physical servers directly interconnected by a local area network.

The virtual-data-center management interface allows provisioning andlaunching of virtual machines with respect to resource pools, virtualdata stores, and virtual networks, so that virtual-data-centeradministrators need not be concerned with the identities ofphysical-data-center components used to execute particular virtualmachines. Furthermore, the virtual-data-center management serverincludes functionality to migrate running virtual machines from onephysical server to another in order to optimally or near optimallymanage resource allocation, provide fault tolerance, and highavailability by migrating virtual machines to most effectively utilizeunderlying physical hardware resources, to replace virtual machinesdisabled by physical hardware problems and failures, and to ensure thatmultiple virtual machines supporting a high-availability virtualappliance are executing on multiple physical computer systems so thatthe services provided by the virtual appliance are continuouslyaccessible, even when one of the multiple virtual appliances becomescompute bound, data-access bound, suspends execution, or fails. Thus,the virtual data center layer of abstraction provides avirtual-data-center abstraction of physical data centers to simplifyprovisioning, launching, and maintenance of virtual machines and virtualappliances as well as to provide high-level, distributed functionalitiesthat involve pooling the resources of individual physical servers andmigrating virtual machines among physical servers to achieve loadbalancing, fault tolerance, and high availability. FIG. 8 illustratesvirtual-machine components of a virtual-data-center management serverand physical servers of a physical data center above which avirtual-data-center interface is provided by the virtual-data-centermanagement server. The virtual-data-center management server 802 and avirtual-data-center database 804 comprise the physical components of themanagement component of the virtual data center. The virtual-data-centermanagement server 802 includes a hardware layer 806 and virtualizationlayer 808, and runs a virtual-data-center management-server virtualmachine 810 above the virtualization layer. Although shown as a singleserver in FIG. 8, the virtual-data-center management server (“VDCmanagement server”) may include two or more physical server computersthat support multiple VDC-management-server virtual appliances. Thevirtual machine 810 includes a management-interface component 812,distributed services 814, core services 816, and a host-managementinterface 818. The management interface is accessed from any of variouscomputers, such as the PC 708 shown in FIG. 7. The management interfaceallows the virtual-data-center administrator to configure a virtual datacenter, provision virtual machines, collect statistics and view logfiles for the virtual data center, and to carry out other, similarmanagement tasks. The host-management interface 818 interfaces tovirtual-data-center agents 824, 825, and 826 that execute as virtualmachines within each of the physical servers of the physical data centerthat is abstracted to a virtual data center by the VDC managementserver.

The distributed services 814 include a distributed-resource schedulerthat assigns virtual machines to execute within particular physicalservers and that migrates virtual machines in order to most effectivelymake use of computational bandwidths, data-storage capacities, andnetwork capacities of the physical data center. The distributed servicesfurther include a high-availability service that replicates and migratesvirtual machines in order to ensure that virtual machines continue toexecute despite problems and failures experienced by physical hardwarecomponents. The distributed services also include a live-virtual-machinemigration service that temporarily halts execution of a virtual machine,encapsulates the virtual machine in an OVF package, transmits the OVFpackage to a different physical server, and restarts the virtual machineon the different physical server from a virtual-machine state recordedwhen execution of the virtual machine was halted. The distributedservices also include a distributed backup service that providescentralized virtual-machine backup and restore.

The core services provided by the VDC management server include hostconfiguration, virtual-machine configuration, virtual-machineprovisioning, generation of virtual-data-center alarms and events,ongoing event logging and statistics collection, a task scheduler, and aresource-management module. Each physical server 820-822 also includes ahost-agent virtual machine 828-830 through which the virtualizationlayer can be accessed via a virtual-infrastructure applicationprogramming interface (“API”). This interface allows a remoteadministrator or user to manage an individual server through theinfrastructure API. The virtual-data-center agents 824-826 accessvirtualization-layer server information through the host agents. Thevirtual-data-center agents are primarily responsible for offloadingcertain of the virtual-data-center management-server functions specificto a particular physical server to that physical server. Thevirtual-data-center agents relay and enforce resource allocations madeby the VDC management server, relay virtual-machine provisioning andconfiguration-change commands to host agents, monitor and collectperformance statistics, alarms, and events communicated to thevirtual-data-center agents by the local host agents through theinterface API, and to carry out other, similar virtual-data-managementtasks.

The virtual-data-center abstraction provides a convenient and efficientlevel of abstraction for exposing the computational resources of acloud-computing facility to cloud-computing-infrastructure users. Acloud-director management server exposes virtual resources of acloud-computing facility to cloud-computing-infrastructure users. Inaddition, the cloud director introduces a multi-tenancy layer ofabstraction, which partitions VDCs into tenant-associated VDCs that caneach be allocated to a particular individual tenant or tenantorganization, both referred to as a “tenant.” A given tenant can beprovided one or more tenant-associated VDCs by a cloud director managingthe multi-tenancy layer of abstraction within a cloud-computingfacility. The cloud services interface (308 in FIG. 3) exposes avirtual-data-center management interface that abstracts the physicaldata center.

FIG. 9 illustrates a cloud-director level of abstraction. In FIG. 9,three different physical data centers 902-904 are shown below planesrepresenting the cloud-director layer of abstraction 906-908. Above theplanes representing the cloud-director level of abstraction,multi-tenant virtual data centers 910-912 are shown. The resources ofthese multi-tenant virtual data centers are securely partitioned inorder to provide secure virtual data centers to multiple tenants, orcloud-services-accessing organizations. For example, acloud-services-provider virtual data center 910 is partitioned into fourdifferent tenant-associated virtual-data centers within a multi-tenantvirtual data center for four different tenants 916-919. Eachmulti-tenant virtual data center is managed by a cloud directorcomprising one or more cloud-director servers 920-922 and associatedcloud-director databases 924-926. Each cloud-director server or serversruns a cloud-director virtual appliance 930 that includes acloud-director management interface 932, a set of cloud-directorservices 934, and a virtual-data-center management-server interface 936.The cloud-director services include an interface and tools forprovisioning multi-tenant virtual data center virtual data centers onbehalf of tenants, tools and interfaces for configuring and managingtenant organizations, tools and services for organization of virtualdata centers and tenant-associated virtual data centers within themulti-tenant virtual data center, services associated with template andmedia catalogs, and provisioning of virtualization networks from anetwork pool. Templates are virtual machines that each contains an OSand/or one or more virtual machines containing applications. A templatemay include much of the detailed contents of virtual machines andvirtual appliances that are encoded within OVF packages, so that thetask of configuring a virtual machine or virtual appliance issignificantly simplified, requiring only deployment of one OVF package.These templates are stored in catalogs within a tenant's virtual-datacenter. These catalogs are used for developing and staging new virtualappliances and published catalogs are used for sharing templates invirtual appliances across organizations. Catalogs may include OS imagesand other information relevant to construction, distribution, andprovisioning of virtual appliances.

Considering FIGS. 7 and 9, the VDC-server and cloud-director layers ofabstraction can be seen, as discussed above, to facilitate employment ofthe virtual-data-center concept within private and public clouds.However, this level of abstraction does not fully facilitate aggregationof single-tenant and multi-tenant virtual data centers intoheterogeneous or homogeneous aggregations of cloud-computing facilities.

FIG. 10 illustrates virtual-cloud-connector nodes (“VCC nodes”) and aVCC server, components of a distributed system that provides multi-cloudaggregation and that includes a cloud-connector server andcloud-connector nodes that cooperate to provide services that aredistributed across multiple clouds. VMware vCloud™ VCC servers and nodesare one example of VCC server and nodes. In FIG. 10, seven differentcloud-computing facilities are illustrated 1002-1008. Cloud-computingfacility 1002 is a private multi-tenant cloud with a cloud director 1010that interfaces to a VDC management server 1012 to provide amulti-tenant private cloud comprising multiple tenant-associated virtualdata centers. The remaining cloud-computing facilities 1003-1008 may beeither public or private cloud-computing facilities and may besingle-tenant virtual data centers, such as virtual data centers 1003and 1006, multi-tenant virtual data centers, such as multi-tenantvirtual data centers 1004 and 1007-1008, or any of various differentkinds of third-party cloud-services facilities, such as third-partycloud-services facility 1005. An additional component, the VCC server1014, acting as a controller is included in the private cloud-computingfacility 1002 and interfaces to a VCC node 1016 that runs as a virtualappliance within the cloud director 1010. A VCC server may also run as avirtual appliance within a VDC management server that manages asingle-tenant private cloud. The VCC server 1014 additionallyinterfaces, through the Internet, to VCC node virtual appliancesexecuting within remote VDC management servers, remote cloud directors,or within the third-party cloud services 1018-1023. The VCC serverprovides a VCC server interface that can be displayed on a local orremote terminal, PC, or other computer system 1026 to allow acloud-aggregation administrator or other user to accessVCC-server-provided aggregate-cloud distributed services. In general,the cloud-computing facilities that together form amultiple-cloud-computing aggregation through distributed servicesprovided by the VCC server and VCC nodes are geographically andoperationally distinct.

Event-Message Generation and Alert Generation and Distribution

FIG. 11 illustrates a simple example of event-message logging andanalysis. In FIG. 11, a number of computer systems 1102-1106 within adistributed computing system are linked together by an electroniccommunications medium 1108 and additionally linked through acommunications bridge/router 1110 to an monitoring-and-management system1112 that includes an administrative console 1114. As indicated bycurved arrows, such as curved arrow 1116, multiple components withineach of the discrete computer systems 1102 and 1106 as well as thecommunications bridge/router 1110 generate event messages which areultimately transmitted to the monitoring-and-management system 1112.Event messages may be relatively directly transmitted from a componentwithin a discrete computer system to the monitoring-and-managementsystem or may be collected at various hierarchical levels within adiscrete computer and then forwarded from an event-message-collectingentity within the discrete computer to the monitoring-and-managementsystem. The monitoring-and-management system 1112 may filter and analyzethe received event messages, as they are received, in order to detectvarious operational anomalies and impending failure conditions. Inaddition, the monitoring-and-management system collects and stores thereceived event messages in one or more data-storage devices orappliances 1118 as entries in indexed files, databases, and/or othertypes of data-storage entities 1120. Either through real-time analysisor through analysis of log files, the monitoring-and-management systemmay detect operational anomalies and conditions for which themonitoring-and-management system displays warnings and informationaldisplays, such as the warning 1122 shown in FIG. 11 displayed on theadministration-computer display device 1114.

FIG. 12 shows a small, 11-entry portion of a log file from a distributedcomputer system. In FIG. 12, each rectangular cell, such as rectangularcell 1202, of the portion of the log file 1204 represents a singlestored event message. In general, event messages are relatively cryptic,including generally only one or two natural-language sentences orphrases as well as various types of file names, path names, and, perhapsmost importantly, various alphanumeric parameters. For example, logentry 1202 includes a short natural-language phrase 1206, date 1208 andtime 1210 parameters, as well as alphanumeric parameter 1212 whichidentifies the particular source of the event.

FIG. 13 illustrates the distribution of alerts from a distributedcomputer system to system-administration-personnel devices. Themonitoring-and-management system 1112, previously discussed withreference to FIG. 11, is shown on the left-hand side of FIG. 13. Inaddition to logging events corresponding to received event messages, themonitoring-and-management system monitors incoming event messages andinitiates generation of alert-messages for certain of the event messagesthat report the occurrences of serious events that need the attention ofsystem-administration personnel. In certain implementations, themonitoring-and-management system itself includes analert-generation-and-distribution subsystem to which theevent-processing system issues alert-initiation requests, while, inother implementations, the monitoring-and-management system generatesalert-initiation messages and transmits the alert-initiation messages toan alert-generation-and-distribution-subsystem component of thedistributed computer system or to a remotealert-generation-and-distribution system that distributes alert messagesto system-administration-personnel devices. As shown in FIG. 13, alertmessages may be distributed to personal computers and workstations1302-1303, cell phones 1304-1305, laptops and tablets 1306, and to manyother types of electronic devices, including pagers and land-linephones. An alert-generation-and-distribution system or subsystem mayinclude complex logic for identifying the appropriate recipients for aparticular alert, identifying to which of the recipients' devices totransmit alert messages, identifying times of day at which to send alertmessages to particular recipients, and identifying the frequency ofalert-message retransmission in the case that alert messages are notreceived in a timely fashion. Alert messages provide rapid notificationto engineers and system administrators of the occurrence of potentiallyserious events that, if not handled in a timely fashion, may lead tosystem degradation or failure. In certain systems, alert messages mayadditionally be sent to automated system-maintenance andsystem-administration subsystems that can undertake corrective andameliorative actions for certain types of reported events without humanintervention.

FIG. 14 illustrates the display of an alert message on asystem-administration-personnel device. The alert message 1402 may bedisplayed in superposition over whatever else is being displayed on thedevice. The alert message may include a date 1404, a time 1405, analert-priority indication 1406, a textural description of the event orproblem that elicited distribution of the alert message 1408, and adisplay-notes feature 1410 to which a user may input an indication todisplay notes associated with the alert message. In the example of FIG.14, user input to the display-notes feature 1410 results in a display ofa list of notes 1412. Each note associated with the alert is summarizedby an indication of the individual who submitted the note 1414, a date1416, and a short title 1418. The list elements are active displayelements. When a user inputs an indication to display a particular noteto an element of the list representing that note, the note is displayedon the device 1420. The note generally includes a comment, a paragraph,or even a short article describing a user's experience with handling thealert, including how the user remediate the system failure or conditionfor which the alert was generated, and other relevant observations thatmay be of use to others. System-administration personnel and other userscan enter notes with respect to particular alerts through analert-system user interface.

FIGS. 15A-B provide control-flow diagrams that illustrate operation ofthe monitoring-and-management system shown in FIGS. 11 and 13. FIG. 15Aprovides a control-flow diagram for an event-message-administrationcomponent of the monitoring-and-management system. In step 1502, theevent-message-administration component initializes various datastructures, file-directory pointers, and event-generation andevent-reception facilities. The event-message-administration componentthen continuously executes an event-handling loop of steps 1504-1517. Instep 1504, the event-message-administration component waits for a nextevent to occur. When the next occurring event is an incoming eventmessage, as determined in step 1505, an event-message handler is calledin step 1506. When the next-occurring event is an event-retrievalrequest received from an event-message-administration user interface, asdetermined in step 1507, an event-retrieval-request handler is called instep 1508. When the next-occurring event is a timer expiration for alog-analysis timer that indicates when automated analysis of event logsis to be next undertaken, as determined in step 1509, anevent-log-analysis handler is called in step 1510. As discussed below,the event-message handler 1506 may generate an alert corresponding to anincoming event message. Similarly, the event-log-analysis handler maygenerate one or more alerts based on analysis of event logs. Alertsgenerated by the event-message handler are near-real-time alerts whilealerts generated by the event-message-analysis handler are delayed byperiods of time of up to the log-analysis timer interval. When thenext-occurring event is a timer expiration for an archiving timer thatindicates when archiving of current event logs is to be next undertaken,as determined in step 1511, an archive handler is called in step 1512.When the next-occurring event is receipt of an event-log query, asdetermined in step 1513, an event-log-query handler is called in step1514. A default handler 1515 handles rare and unexpected events. Whenthere are more events queued for handling, as determined in step 1516, anext event is dequeued, in step 1517, and control returns to step 1505.Otherwise, control returns to step 1504, where theevent-message-administration component waits for a next event to occur.Ellipses 1518 and 1519 indicate that many other types of events may bereceived and handled by the event-handling loop of steps 1504-1513.

FIG. 15B provides a control-flow diagram for the event-message handlercalled in step 1506 of FIG. 15A. In step 1520, the event-message handlerreceives the event message. In step 1521, the event-message handlerparses the received event message and identifies any available relatedinformation that may be useful in analyzes the event message. In step1522, the event-message handler analyzes the event message, along withany of the identified related information, to determine anyadministrative actions to undertake as a result of the occurrence of theevent corresponding to the event message. For example, the event-messagehandler may extract an event type from the received event message andlook up the event type in an alert list that contains alert-list entriesfor event-message types for which alerts are generated. When the type ofthe received event message is found in the alert list, as determined instep 1524, the event-message handler, in step 1526 extracts informationfrom the received event message and the corresponding alert-list entryand creates an alert-initiation message that contains informationincluding an indication of an alert type, source of the alert, date andtime that the alert-inducing event occurred, and other such information.In step 1528, the alert-initiation message is forwarded to analert-generation-and-distribution system or subsystem for distributionto system-administration personnel. Other types of administrativeactions may be taken, depending on the event type and otheradministration parameters. In step 1530, the event-message handlerprocesses the received event message to create an event-store entry and,in step 1532, the event-message handler writes the entry to the eventstore. Thus, as previously mentioned, in many distributed computingsystems, alerts are generated, by an alert-generation-and-distributionsystem or subsystem, in response to receiving alert-initiation messages,by the alert-generation-and-distribution system or subsystem, from anmonitoring-and-management system that transmits alert-initiationmessages in response to receiving certain types of event messages. Thealert-generation-and-distribution system may be component of thedistributed computer system or a remoted system external to thedistributed computing system. As mentioned above, alerts may also begenerated during periodic analysis of recently stored event-storeentries.

Currently available alert-generation-and-distribution systems andsubsystems greatly facilitate administration of distributed computersystems, including large data centers. However, currently availablealert-generation-and-distribution systems and subsystems have numerousdeficiencies. Many of the currently availablealert-generation-and-distribution systems are inflexible andinefficient. For example, it is often the case that particular alertsmay be generated erroneously, due to programming orevent-message-processing errors, may include descriptions that do notwell correspond to the underlying events and problems for which thealert is generated, and may be associated with incorrect priorities. Thepriority associated with an alert may determine when alert messages aresent, to whom alert messages are sent, the frequency of retransmissionof alert messages for which no response is received, the device types towhich the alerts are sent, and the times of day during which the alertmessages are transmitted. When an alert is associated with a higherpriority than warranted, occurrence of the alert may result indistraction of already busy and preoccupied system administrators andengineers and/or in significant amounts of wasted effort and time whensystem administrators, engineers, and other personnel are diverted fromtheir current tasks to diagnose and to attempt to ameliorate problemsthought to be associated with the occurrence of the alert. Furthermore,large distributed computer systems, including large physical datacenters, are highly dynamic entities that produce gigabytes and eventerabytes of log-entry data on a daily basis. Analert-generation-and-distribution system that needs to be reprogrammedor manually reconfigured in order to respond to changing patterns ofsystem behavior is hopelessly unresponsive and fragile in the face ofthe potential scale and speed of system changes that can render thealert-generation-and-distribution system outdated and uninformative.

Collaborative Alert-Generation-and-Distribution Systems to which theCurrent Document is Directed

The current document discloses a new, flexible, and robustalert-generation-and-distribution system or subsystem that employs userfeedback, user-initiated proposals for modifying thealert-generation-and-distribution system, and consensus-basedmodification-proposal adoption in order to quickly, precisely, andcontinuously adjust the alert-generation-and-distribution system totrack the behavior and operational characteristics of a distributedcomputer system for which alerts are generated. In essence, thecurrently disclosed alert-generation-and-distribution system provides asocial-networking-like collaborative environment in which systemadministrators, engineers, and other personnel cooperate to continuouslyadapt the alert-generation-and-distribution system to current systemconditions and characteristics as well as to the current collectiveunderstanding of system conditions and characteristics within thecollaborative environment.

FIGS. 16A-B illustrate, from a user-interface perspective, certainfeatures of the currently disclosed, collaborativealert-generation-and-distribution system or subsystem. In FIGS. 16A-B, aseries of display-screen captures, similar to those shown in FIG. 14,are used to illustrate these features. The currently disclosedcollaborative alert-generation-and-distribution system provides a userinterface (“UI”) that can be invoked from any of many different types ofuser devices. Screen capture 1602 in FIG. 16A shows an example landingpage or landing screen for thecollaborative-alert-generation-and-distribution-system UI. The UIprovides various options to the user, including: (1) an option 1604 toupdate personal data, such as the user's name, position, user devices,user-device network addresses or other access codes, and other personalinformation; (2) an option 1606 to submit a proposal to modify thecollaborative alert-generation-and-distribution system; (3) an option1608 to view submitted modification proposals; (4) an option 1610 forvoting for or against one or more proposals; (5) an option 1612 toaccess an administration user interface by authorized administrators;(6) an option 1614 to add a note to a particular type of alert; (7) anoption 1616 to view notes previously submitted for association withparticular types of alerts; and (8) an option 1618 to view a list of thevarious different types of alerts that can be generated by thecollaborative alert-generation-and-distribution system. Of course, inany particular implementation, the UI may include many additional and/ordifferent options.

Screen capture 1620 illustrates the UI page displayed to a user whoselects the “submit proposal” option 1606 from the landing-page menu.The “submit proposal” page lists numerous different types ofmodifications to the collaborative alert-generation-and-distributionsystem that can be proposed by a user, including: (1) changing thecurrent status of a particular alert 1622; (2) changing the name of analert 1623; (3) changing the description of an alert 1624; (4) changingthe base priority for an alert 1625; (5) changing a mode associated withan alert 1626; (6) adding a source for an alert 1627; (7) deleting asource from an alert 1628; (8) updating, or editing, a source for analert 1629; (9) adding a recipient for an alert 1630; (10) deleting arecipient for an alert 1631; (11) adding a new device type 1632; (12)deleting a device type 1633; (13) updating a device type 1634; and (14)updating a user rating 1635. In any given implementation, there may bemany different and/or additional proposed modification types. An inputfeature 1636 is provided to view additional proposed modification typesin the example UI shown in FIG. 16A. Of course, in any particularimplementation, the above-discussed UI captured may be differentlyorganized. For example, relevant change features, such as the “ChangeAlert Status” feature, may be arranged near relevant editable objects,such as an editable display window that displays the current alertstatus. Alternatively, the change features may be displayed in responseto user input to relevant display features.

Screen capture 1640 illustrates a “change alert status” page displayedwhen a user selects the “change alert status” option 1622 displayed inthe “submit proposal” UI page. The “change alert status” page includesan input feature 1642 that allows the user to specify the name of analert, after which the “change alert status” page displays the currentstatus for the alert 1644. A second input feature 1646 allows a user tosubmit a proposed new status for the alert, with the new statusselectable from a status-selection window 1648. When the user inputs aselection-indication to the “next” input feature 1649 of the “changealert status” page, a “new proposal” page is displayed, as shown inscreen capture 1650 in FIG. 16B. The “new proposal” page displaysinformation 1652 associated with the status-modification proposal andprovides a text-input feature 1654 that allows the user to include adescription of the status-modification proposal. For those proposalsneeding one or more parameter values, discussed further below,parameter-value-specification prompts may be displayed within thetext-input feature to prompt the user to supply the needed parametervalues. A “submit” input feature 1656 allows a user to submit thefinished proposal to the collaborative alert-generation-and-distributionsystem or subsystem. User input to the “submit” input feature results ina display of a summary of the proposed modification to the user, asshown in screen capture 1660 in FIG. 16B.

Once a proposed modification has been successfully submitted, thecollaborative alert-generation-and-distribution system transmits, incertain implementations, vote-request messages to all or a subset of theusers or participants of the collaborativealert-generation-and-distribution system. Screen capture 1670 in FIG.16B shows an example “vote” page displayed by the collaborativealert-generation-and-distribution system to a user on the user's devicefollowing reception, by the device, of a vote-request message. The“vote” page displays information about the proposed modification 1672and provides a “description” input feature 1674 to allow a user todisplay the description for the proposed modification in an additionalscreen. Two voting input features 1676 and 1678 allow a user to input avote for or against the proposed modification. When votes are submitted,as further discussed below, the collaborativealert-generation-and-distribution system, depending on a current modefor the alert for which the modification is submitted or, for non-alertmodifications, a modification mode associated with the proposedmodification, either automatically carries out the proposedmodification, when the submitted votes indicate a consensus amongalert-generation-and-distribution-system participants in favor of themodification, or discards the proposed modification, when the submittedvotes indicate a lack of consensus in favor of the modification. Incertain cases, the mode associated with an alert or a proposedmodification may indicate that, rather than automatically accepting aproposal, the collaborative alert-generation-and-distribution systemforwards a user-accepted proposal to a system administrator or otherindividual or group of individuals for final authorization andimplementation. Vote-request messages may be sent out immediately,following submission of a proposal, or may be sent out atparameter-specified times. Different subsets of users may receivevote-request messages for different types of proposed modifications orfor proposed modifications to different alerts. For example, in certainimplementations, only those users who are designated as before/aftermodification recipients for a particular alert may be requested to voteon proposed modifications to that alert. When a sufficient number ofresponses to a vote-request message are not yet received, thecollaborative alert-generation-and-distribution system may retransmitthe vote-request message to unresponsive users. In addition, users maysubmit votes independently of receiving vote-request messages throughthe UI (1610 in FIG. 16A).

Next, an implementation of the collaborativealert-generation-and-distribution system or subsystem to which thecurrent document is directed can be described using arelational-database schema and control-flow diagrams. This is, ofcourse, an example implementation. Alternative implementations can useother types of databases or data-storage functionalities. FIGS. 17A-Bshow a number of relational-database tables that represent the storeddata that is maintained by the collaborativealert-generation-and-distribution system and that is used for alertdistribution, proposed-modification submission, voting for or againstproposed modifications, and other functionalities and facilitiesprovided by the collaborative alert-generation-and-distribution system.FIG. 17A shows representations of seven relational-database tables.These tables include: (1) ALERTS 1702, each row of which represents adifferent alert generated and distributed by the collaborativealert-generation-and-distribution system; (2) ALERT_NOTES 1703, each rowof which represents the association of a particular note with aparticular type of alert; (3) NOTES 1704, each row of which represents anote associated with one or more alerts; (4) ALERT_GROUPS 1705, each rowof which represents the association of a particular type of alert with aparticular alert group; (5) GROUPS 1706, each row of which represents agroup of alerts; (6) SOURCES 1707, each row of which represents acomponent, subcomponent, or subsystem of a distributed computer systemthat initiates events that are communicated to, and collected by, themonitoring-and-management system and that result in alert initiation;(7) ALERT_SOURCE_PRIORITIES 1708, each row of which represents apriority-modification term applied to the base priority of a particularalert for a particular source of the alert. FIG. 17B showsrepresentations of six additional relational-database tables, including:(1) PERSONNEL 1709, each row of which represents a user or participantin the collaborative alert-generation-and-distribution system orsubsystem; (2) DEVICES 1710, each row of which represents a type of theuser device; (3) PERSONNEL_DEVICES 1711, each row of which represents aparticular user device owned by, or associated with, a particularparticipant; (4) RECIPIENTS 1712, each row of which represents aparticipant who receives a particular type of alert; (5) PROPOSALS 1713,each row of which represents a proposal for modifying thealert-generation-and-distribution system; and (6) VOTES 1714, each rowof which represents a vote, by a participant, for or against aparticular proposal. The tables shown in FIGS. 17A-B represent a subsetof the information generally stored and maintained by the currentlydisclosed collaborative alert-generation-and-distribution system. Thissubset of information is sufficient to describe, below, theimplementation of significant features of the currently disclosedcollaborative alert-generation-and-distribution system.

The table ALERTS includes the following columns: (1) Alert_Type 1716, aunique identifier for each alert represented by a row in the tableALERTS; (2) Status 1717, the current status for an alert, which mayinclude enabled, disabled, and temporary maintenance; (3) Name 1718, anatural-language name for the alert; (4) Description 1719, a texturaldescription of the alert, including parameter values for an alertassociated with one or more parameters, discussed below; (5)Base_Priority 1720, the base priority for the alert; (6) Change_Mode1721, an indication of how the various values associated with the alert,including the values represented by columns in the table alerts, aremodified; (7) Creator_ID 1722, the unique identifier for a participantthat created the alert; and (8) Creation_Date_Time 1723, an indicationof the date and time when the alert type was created. The broken cell1724 indicates that there may be additional columns in any particularimplementation of the ALERTS table. Of course, in differentimplementations of the collaborative alert-generation-and-distributionsystem, any of the currently discussed relational-database tables mayhave fewer, more, and/or different columns, and the collaborativealert-generation-and-distribution system may use fewer, more, ordifferent tables.

The table NOTES includes the following columns: (1) Note_ID 1726, aunique identifier for the note represented by a row in the table NOTES;(2) Document_Path 1727, a file-system path for a document that containsthe note; (3) Priority 1728, a display priority associated with the notethat determines the order in which the note is listed when the notesassociated with an alert are displayed to a user (1412 in FIG. 14); (4)Creator_ID 1729, the unique identifier for the participant who createdthe note; and (5) Creation_Data_Time 1730, the data and time when thenote was created.

The table PERSONNEL includes the following columns: (1) P_ID 1732, aunique identifier for a participant or user represented by a row of thetable PERSONNEL; (2) First_Name 1733, the first name of the participant;(3) Last_Name 1734, the last name of the participant; (4) Address 1735,an address for the participant; (5) Position 1736, the participant's injob title; (6) Rating 1737, a participant rating reflective of theconsensus value of the participant's notes, modification proposals, andother collaborative activities; and (7) Job_Description 1738, a shortdescription of the participant's role in the collaborativealert-generation-and-distribution system.

The table DEVICES includes the following columns: (1) Device_Type 1740,a unique identifier for a device type represented by a row in the tableDEVICES; (2) Access_Method 1741, an indication of how the device isaccessed; and (3) Display_Parameters 1742, a list of one or more displayparameters that control generation and formatting of UI pages and otherinformation sent to the device by the alert-generation-and-distributionsystem.

The table PERSONNEL_DEVICES includes the following columns: (1) P_ID1744, the identifier for the participant owner of the device; (2)Device_Type 1745, an identifier for the type of the device; (3)Address/Number 1746, the IP address, telephone number, or othercommunications address used to access the device; (4) Access_Code 1747,an access code that may be needed to access the device; and (5) Priority1748, an indication of the priority for accessing the particular deviceamong all of the devices by which a particular user can be reached.

The table PROPOSALS includes the following columns: (1) Proposal_ID1715, a unique identifier for the modification proposal represented by arow in the table PROPOSALS; (2) Submission_Date_Time 1751, a date andtime when the proposal was submitted; (3) Submitter_ID 1752, a uniqueidentifier for the participant who submitted the proposal; (4)Change_Type 1753, the type of the proposal, including the types listedin screen capture 1620 shown in FIG. 16A; and (5) Change_Description1754, the description furnished in the description input window 1654shown in screen capture 1650 of FIG. 16B. The meanings of the columns ofthe remaining tables are either self-evident from the column names orunneeded in the following discussion.

Relational database tables are accessed and manipulated using a querylanguage, such as the structured query language (“SQL”). FIG. 18provides a number of examples of SQL queries in the context of therelational database tables shown in FIGS. 17A-B. The create-table query1802 creates the GROUP_ID table (1706 in FIG. 17 A). The select query1804 retrieves the name and description from the ALERTS table for analert with unique alert type equal to 361. The select query 1806retrieves the value of the Priority_mult field from theALERT_SOURCE_PRIORITIES table and the Name field from the ALERTS tablefor the alert with alert type 361. This is an example of a join querywhich involves logically combining the two tables ALERTS andALERT_SOURCE_PRIORITIES in order to retrieve the desired information.The insert query 1808 inserts a new row into the VOTES table. Thecreate-procedure query 1810 creates a routine, or stored procedure,Vote, that is stored within the database management system thatmaintains the relational database tables and that supports queryprocessing for accessing and manipulating the relational databasetables. The Vote routine takes, as arguments, an integer parameter 1812and a Boolean parameter 1814 and uses the input parameter values tocreate and insert a new row into the Votes table. The execute query 1816invokes the stored procedure Vote, created by the create-procedure query1810, to add a new row to the VOTES table with field valuesProposal_ID=4621, P_ID=13, and Vote=TRUE. The delete query 1818 deletesall rows from the VOTES table with field value Proposal_ID=4621. Theupdate-alerts query 1820 changes the value of the field Description inthe row of the ALERTS table with field value Alert_Type=203. Thus, SQLqueries can be used to create and delete tables, create and deletestored procedures, retrieve information from tables, change informationstored in tables, add and delete rows from tables, and carry out otherdata-retrieval and data-manipulation operations. In the control-flowdiagrams discussed below, it is assumed that access to storedinformation within the relational database tables maintained by thecollaborative alert-generation-and-distribution system is carried outvia SQL queries. Of course, any of many other data-storage anddata-manipulation methodologies can be used in place of a relationaldatabase system and SQL queries in alternative implementations of thecollaborative alert-generation-and-distribution system.

FIGS. 19A-F provide control-flow diagrams that illustrate portions of animplementation of the currently disclosed collaborativealert-generation-and-distribution system. FIG. 19A provides acontrol-flow diagram for an event loop that underlies the describedimplementation of the collaborative alert-generation-and-distributionsystem. In step 1902, the database system used to store theabove-discussed relational database tables is initialized orreinitialized, where reinitialization involves launching the databasemanagement system and verifying that the above-discussed tables havebeen created. Then, the collaborative alert-generation-and-distributionsystem continuously executes the event-handling loop of steps 1903-1914.In step 1903, the collaborative alert-generation-and-distribution systemwaits for the occurrence of a next event. When the next-occurring eventrepresents reception of an alert-initiation message, as determined instep 1904, a generate-alerts event handler is called, in step 1905. Whenthe next-occurring event is a reception of a vote for or against aproposed modification, as determined in step 1906, a vote event handleris called in step 1907. When the next-occurring event is a selection,through the UI, for the option of submitting a new proposed modificationand collection of information for the proposed modification through theUI, as determined in step 1908, a new-proposal event handler is called,in step 1909. When the next-occurring event is a request, received froma user device, to display the UI home page, as determined in step 1910,a participant-UI event handler is called in step 1911. Ellipses 1916indicate that many other types of events may be handled by the eventloop. A default handler 1912 handles any unexpected or rare events. Whenmore events have been queued for handling, as determined in step 1913, anext event is dequeued, in step 1914, and control returns to step 1904.Otherwise, control returns to step 1903, where the collaborativealert-generation-and-distribution system waits for a next event tooccur.

FIG. 19B provides a control-flow diagram for the generate-alert eventhandler, called in step 1905 of FIG. 19A. In step 1920, thegenerate-alert handler extracts an alert type from the receivedalert-initiation message. As discussed above, alert-initiation messagesare received from the previously discussed monitoring-and-managementsystem that receives and handles event messages from components withinthe distributed computer system and that transmits alert-initiationmessages for certain of the received events to the collaborativealert-generation-and-distribution system. In step 1921, thegenerate-alert handler selects a row r from the ALERTS table with thefield value r.Alert_Type equal to the alert type extracted from thereceived alert message. When a row r is found by the select statement,as determined in step 1922, and when the field r.Status of row r has avalue enabled, as determined in step 1923, the generate-alert handlerextracts any additional information included in the alert-initiationmessage and uses this additional information, along with the values ofthe fields of row r, to prepare an alert for distribution toparticipants, in step 1924. In step 1925, the generate-alert handlerattempts to select a row from the table ALERT_NOTES with a field valueAlert_Type equal to the alert type extracted from the received alertmessage. When a row is found in the ALERT_NOTES table, as determined instep 1926, a display-notes feature (1410 in FIG. 14) is added to thealert, in step 1927. The handler generate-alert then calls the routine“distribute alert,” in step 1928, to distribute the alert tocollaborative-alert-generation-and-distribution-system participants andreturns, in step 1929. When there is no row in the ALERTS tablecorresponding to the extracted alert type or when the status of thealert is not enabled, as determined in steps 1922 and 1923, thegenerate-alert handler returns, in step 19230

FIG. 19C provides a control-flow diagram for the routine “distributealert,” called in step 1928 of FIG. 19B. In step 1930, the routine“distribute alert” selects a row s from the tableALERT_SOURCE_PRIORITIES with the field value s.Alert_Type equal to thealert type extracted from the received alert message, in step 1920 ofFIG. 19B, and with field value s.Source_ID equal to an indication of thesource of the alert message extracted from the alert message in step1924 of FIG. 19B. When a row s found, as determined in step 1931, theroutine “distribute alert,” in step 1932, sets the priority of the alertto the base priority for the alert, stored in the field r.Base_Priority,multiplied by the value of the field s.Priority_mult. Otherwise, in step1933, the priority of the alert is set to the base priority and fieldr.base_priority. In step 1934, the distributed-alert handler selects adistribution queue q onto which to queue the alert for distribution.Selection of the distribution queue is based on the priority assigned tothe alert in step 1932 or 1933. In step 1935, the routine“distribute-alert” uses a join-type select query, involving a join ofthe tables RECIPIENTS, PERSONNEL, DEVICES, and PERSONNEL_DEVICES, toobtain a list of devices and associated display parameters to which tosend the alert. In the for-loop of steps 1936-1938, thedistributed-alert routine queues a version of the alert in an alertmessage to the output queue q selected in step 1934 for each device inthe list of devices. Each alert message contains alert information thatis tailored for display on a particular device, using the displayparameters for that device obtained in step 1935.

FIG. 19D provides a control-flow diagram for the vote event handlercalled in step 1907 of FIG. 19A. In step 1940, the vote handler receivesa vote message sent by a participant. In step 1941, the vote handlerselects a row p from the PERSONNEL_DEVICES table with the field valueAddress/Number equal to the device address of the device from which thevote message was received. When a row p is not found, as determined instep 1942, an error-handling routine is called, in step 1943. This errormay occur, for example, when information about a new device has not yetbeen entered into the PERSONNEL_DEVICES table. When the error issuccessfully handled, as determined in step 1944, control returns tostep 1941 to again attempt to select a row p from the tablePERSONNEL_DEVICES. When the error is not successfully handled, the votehandler returns, in step 1945. When a row p is found, the vote handlerextracts the proposal identifier and the user's unique identifier, instep 1946, from the vote message received in step 1940. In step 1947,the vote handler attempts to select a row v from the VOTES table withfield value Proposal_ID equal to the extracted proposal identifier andwith the field P_ID equal to the participant's identifier extracted fromthe received vote message. If a row v is found, as determined in step1948, an error handling routine is called, in step 1949, to handle thefact that a participant appears to have attempted to vote two or moretimes for the proposal. Otherwise, in step 1950, the vote handlerextracts a vote from the vote message and, in step 1951, inserts a newrow in the VOTES table corresponding to the extracted vote. In step1952, the vote handler uses a select statement to count the number ofrows nv in the VOTES table that have the field value Proposal_ID equalto the extracted proposal identifier. In step 1953, the vote handleruses a select statement to count the number of rows np in the PERSONNELtable or the number of rows np in a subset of the rows in the PERSONNELtable representing those users who are designated as respondents to theproposal. When nv is greater than np multiplied by a thresholdfractional value, as determined in step 1954, a sufficient number ofvotes have been accumulated for the proposed modification to determinewhether or not to implement the proposed modification by calling aroutine “elect proposal,” in step 1955. Otherwise, the vote handlerreturns, in step 1956. The currently described implementation assumesthat all participants vote for all proposals.

FIG. 19E provides a control-flow diagram for the routine “electproposal,” called in step 1955 of FIG. 19D. The routine “elect proposal”may also be called from the event-handling loop of steps 1903-1914 inFIG. 19A in response to an occurrence of a proposal-timeout timerexpiration. In step 1960, the routine “elect proposal” receives aproposal identifier and, optionally, the value np determined in step1953 of FIG. 19D. In step 1961, the routine “elect proposal” uses aselect statement to count the number of rows y in the VOTES table withfield values Proposal_ID equal to the received proposal identifier andwith field value Vote equal to TRUE. In step 1962, the routine “electproposal” counts the number of rows n in the VOTES table with the fieldvalue Proposal_ID equal to the received proposal identifier and with thefield value Vote equal to FALSE. In step 1963, the routine “electproposal” computes the total number of votes for the proposal, t, as thesum of y and n. In addition, the ratio r of “yes” votes to total votesis computed. When the argument np has not been passed to the routine“elect proposal,” as determined in step 1964, a local variable np is setto the number of rows in the PERSONNEL table, in step 1965. A fraction fof participants who have submitted votes for the proposal is determined,in step 1966. When the fraction f is greater than a quorum fraction, asdetermined in step 1967, and when the ratio r is greater than or equalto a ratio that defines a majority, as determined in step 1968, theproposal has received a sufficient number of “yes” votes to represent aconsensus of the participants for making the proposed modification.Otherwise, the proposal has failed. When the proposal has succeeded, theroutine “elect proposal,” in step 1969, selects a row p in the PROPOSALStable with the field value p.Proposals equal to the received proposalidentifier. The field value p.Change_Type in the selected row p is usedto identify a stored procedure sp, in step 1970. In step 1971,information in the Change_Description field of the row p is used toaccess any parameters included in the description of the proposal neededfor calling the stored procedure sp. In step 1972, the stored proceduresp is called with the parameter values obtained, in step 1971, to effectthe proposed modification. Proposed modifications are carried out, inthe described implementation, using stored procedures, with eachdifferent type of proposed modification associated with a storedprocedure that carries out the modification. Finally, in step 1973, rowsin the VOTES tables corresponding to votes for the currently consideredproposal and the row in the table PROPOSALS representing the currentlyconsidered proposal are deleted if not needed for audit purposes.

FIG. 19F provides a control-flow diagram for the handler “new proposal,”called in step 1911 of The FIG. 19A. In step 1980, the handler “newproposal” receives collected proposal information from the UI. In step1981, the new-proposal handler generates a next proposal identifier. Instep 1982, the new-proposal handler prepares a new row p for theproposals table with field value Proposal_ID equal to the proposalidentifier included in the received collected proposal information fromthe UI. In step 1983, the new-proposal handler sets theSubmission_Data_Time and Submitter_ID field values for the row p toinformation extracted from the collected proposal information. In step1984, the Change_Type field for the row p is set to an indication of thetype of the proposed modification. In step 1985, the Change_Descriptionfield of new row p is set to a description extracted from the collectedproposal information that includes any needed parameter values forcalling the corresponding stored procedure that effects the proposedmodification when it is determined that the proposal has succeeded. Instep 1986, the new row p is inserted into the table PROPOSALS. In step1987, a list of devices to which to send a vote request to solicit voteson the currently considered proposal is prepared via a select statementthat uses a join on the PERSONNEL, DEVICES, and PERSONAL_DEVICES tables.In step 1988, a vote-request message is prepared using information inthe row p of the PROPOSALS table and any other relevant informationcontained in other of the relational database tables. Finally, in thefor-loop of steps 1989-1971, the vote message is tailored for eachdevice and queued to an output queue for transmission.

As mentioned above, there are many possible implementation variationsfor the currently disclosed collaborativealert-generations-and-distribution system. More sophisticatedconsensus-determination methodologies may be used in the case that onlysubsets of participants vote for particular proposals, or in case thatparticipant votes are given different weights. More sophisticateddevice-list-determination methods are needed in such cases, as well. Inmany implementations, consensus determination automatically results inproposed-modification acceptance and implementation, while, in otherimplementations, only a subset of the possible proposed modificationsmay be automatically implemented, with others implemented only by asystem administrator or other authorized individual. As noted, above,alerts may be grouped into alert groups, and the group to which an alertbelongs may determine how proposed modifications are accepted andimplemented, to whom vote requests are distributed, and who can proposemodifications to alert parameters.

Although the present invention has been described in terms of particularembodiments, it is not intended that the invention be limited to theseembodiments. Modification within the spirit of the invention will beapparent to those skilled in the art. For example, any of many differentalternative implementations can be obtained by varying any of manydifferent design and implementation parameters, including choice ofhardware components and configurations of distributed computer systems,choice of programming languages, operating systems, virtualizationlayers, control structures, data structures, modular organization, andother such design and implementation parameters. While the currentlydisclosed implementation uses relational database for storing relevantdata, formatted files, non-relational databases, and other data-storagetechniques may be instead use in alternative implementations. Manyadditional sophisticated features may be included in a collaborativealert-generation-and-distribution system or subsystem. For example, acollaborative alert-generation-and-distribution may additionally allowparticipants to define new alerts, including specification onevent-message types corresponding to the alert.

It is appreciated that the previous description of the disclosedembodiments is provided to enable any person skilled in the art to makeor use the present disclosure. Various modifications to theseembodiments will be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherembodiments without departing from the spirit or scope of thedisclosure. Thus, the present disclosure is not intended to be limitedto the embodiments shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

1-20. (canceled)
 21. A system for modifying analert-generation-and-distribution system, comprising: a non-transitory,computer-readable medium comprising instructions; and at least onehardware-based processor that executes the instructions to carry outstages comprising: displaying, on a first graphical user interface (GUI)of a first user device, a plurality of categories for which a proposalrelating to the alert-generation-and-distribution system can besubmitted; in response to a selection of one of the plurality ofcategories, displaying a field for inputting the proposal on the firstGUI; in response to the proposal being submitted through the first GUI,displaying, on a second GUI of a second user device, informationrelating to the proposal and a voting element for indicating whether asecond user of the second user device agrees with the proposal; andexecuting the proposal based on the second user agreeing to the proposalthrough the voting element.
 22. The system of claim 21, whereinexecuting the proposal is further based on the second user's agreementcontributing to a consensus on the proposal among a plurality of users.23. The system of claim 21, wherein the proposal comprises a uniqueidentifier associated with a unique modification procedure.
 24. Thesystem of claim 23, wherein the proposal is executed by executing theunique modification procedure.
 25. The system of claim 21, wherein theplurality of categories comprises at least one of changing an alertstatus, name, description, priority, or mode, and adding, deleting, ormodifying an alert source, alert recipient, or device type.
 26. Thesystem of claim 21, wherein the first GUI includes a plurality ofproposed changes from which a user can choose.
 27. The system of claim21, wherein the second GUI includes a rating associated with a firstuser of the first user device.
 28. A method for modifying analert-generation-and-distribution system, comprising: displaying, on afirst graphical user interface (GUI) of a first user device, a pluralityof categories for which a proposal relating to thealert-generation-and-distribution system can be submitted; displaying,on the first GUI in response to a selection of one of the plurality ofcategories, a field for inputting the proposal; receiving the proposalthrough the first GUI; displaying, on a second GUI of a second userdevice, information relating to the proposal and a voting element forindicating whether a second user of the second user device agrees withthe proposal, wherein the proposal is executed based on the second useragreeing to the proposal through the voting element.
 29. The method ofclaim 28, wherein the proposal is executed based on the second user'sagreement contributing to a consensus on the proposal among a pluralityof users.
 30. The method of claim 28, wherein the proposal comprises aunique identifier associated with a unique modification procedure. 31.The method of claim 30, wherein the proposal is executed by executingthe unique modification procedure.
 32. The method of claim 28, whereinthe plurality of categories comprises at least one of changing an alertstatus, name, description, priority, or mode, and adding, deleting, ormodifying an alert source, alert recipient, or device type.
 33. Themethod of claim 28, wherein the first GUI includes a plurality ofproposed changes from which a user can choose.
 34. The method of claim28, wherein the second GUI includes a rating associated with a firstuser of the first user device.
 35. A non-transitory, computer-readablemedium containing instructions that, when executed by a hardware-basedprocessor, performs stages for modifying analert-generation-and-distribution system, the stages comprising:displaying, on a first graphical user interface (GUI) of a first userdevice, a plurality of categories for which a proposal relating to thealert-generation-and-distribution system can be submitted; in responseto a selection of one of the plurality of categories, displaying a fieldfor inputting the proposal on the first GUI; in response to the proposalbeing submitted through the first GUI, displaying, on a second GUI of asecond user device, information relating to the proposal and a votingelement for indicating whether a second user of the second user deviceagrees with the proposal; and executing the proposal based on the seconduser agreeing to the proposal through the voting element.
 36. Thenon-transitory, computer-readable medium of claim 35, wherein executingthe proposal is further based on the second user's agreementcontributing to a consensus on the proposal among a plurality of users.37. The non-transitory, computer-readable medium of claim 35, whereinthe proposal comprises a unique identifier associated with a uniquemodification procedure.
 38. The non-transitory, computer-readable mediumof claim 37, wherein the proposal is executed by executing the uniquemodification procedure.
 39. The non-transitory, computer-readable mediumof claim 35, wherein the first GUI includes a plurality of proposedchanges from which a user can choose.
 40. The non-transitory,computer-readable medium of claim 35, wherein the second GUI includes arating associated with a first user of the first user device.